Walkthrough

Replaces unbounded, timeout-driven generation KV transfer completion in CacheTransceiver::checkGenTransferStatus with a bounded polling loop using a new allgatherIntValue helper for rank-consistent completion. Adds a centralized _cancel_timed_out_gen_transfers helper in Python, removes the old inline timeout branch from _handle_responses, fixes teardown ordering and shutdown-path guards in sender/receiver threads, and unsets UCX_NET_DEVICES alongside UCX_TLS in disaggregated launch scripts.

Changes

Disaggregated KV Transfer Timeout & Polling Rework

UCX Environment Cleanup Fix

Sequence Diagram(s)

sequenceDiagram
  participant Exec as PyExecutor
  participant Cancel as _cancel_timed_out_gen_transfers
  participant TP as TP Collective
  participant Transceiver as kv_cache_transceiver (C++)
  participant Poll as checkGenTransferStatus

  rect rgba(100, 149, 237, 0.5)
    note over Exec,Poll: Per-iteration disagg gen transfer check
    Exec->>Poll: checkGenTransferStatus()
    Poll->>Poll: allgatherIntValue (rank-align ready count)
    loop up to poll budget
      Poll->>Poll: check futures (wait_for 0ms)
      Poll->>Poll: sleep if not ready
    end
    Poll-->>Exec: completed requests erased from mRequesterFutures
  end

  rect rgba(255, 140, 0, 0.5)
    note over Exec,Cancel: Centralized timeout cancellation (new)
    Exec->>Cancel: _cancel_timed_out_gen_transfers()
    Cancel->>TP: allreduce timed-out request IDs
    TP-->>Cancel: aligned set across TP ranks
    Cancel->>Transceiver: cancel_request(request)
    Cancel->>Cancel: mark DISAGG_TRANS_ERROR
    Cancel->>Exec: _handle_errors(charge_budget=False)
  end

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#15139: Directly related — implements consensus-based deferral of generation transfer terminal-state transitions that this PR's bounded polling and allgather coordination builds upon.

Suggested reviewers

• Shixiaowei02
• pcastonguay
• joyang-nv
• Tabrizian
• bo-nv

❌ Failed checks (1 warning)

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.